Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: add data connectors #407

Merged
merged 11 commits into from
Oct 17, 2024
Merged

feat!: add data connectors #407

merged 11 commits into from
Oct 17, 2024

Conversation

@leafty leafty force-pushed the leafty/build-data-connectors branch from 9a7f277 to 093d7ef Compare September 13, 2024 11:31
@coveralls
Copy link

coveralls commented Sep 13, 2024

Pull Request Test Coverage Report for Build 11288123511

Details

  • 1226 of 1325 (92.53%) changed or added relevant lines in 28 files are covered.
  • 3 unchanged lines in 3 files lost coverage.
  • Overall coverage increased (+0.2%) to 90.667%

Changes Missing Coverage Covered Lines Changed/Added Lines %
components/renku_data_services/data_connectors/core.py 42 43 97.67%
components/renku_data_services/data_connectors/models.py 93 94 98.94%
components/renku_data_services/data_connectors/orm.py 84 85 98.82%
components/renku_data_services/project/orm.py 7 8 87.5%
components/renku_data_services/data_connectors/blueprints.py 139 141 98.58%
components/renku_data_services/migrations/versions/a11752a5afba_migrate_to_entity_slugs.py 62 65 95.38%
bases/renku_data_services/background_jobs/config.py 3 7 42.86%
bases/renku_data_services/background_jobs/core.py 14 21 66.67%
components/renku_data_services/authz/schemas.py 12 19 63.16%
components/renku_data_services/authz/authz.py 138 147 93.88%
Files with Coverage Reduction New Missed Lines %
bases/renku_data_services/background_jobs/config.py 1 53.45%
components/renku_data_services/base_api/auth.py 1 89.61%
components/renku_data_services/storage/apispec_base.py 1 93.75%
Totals Coverage Status
Change from base Build 11257919430: 0.2%
Covered Lines: 10589
Relevant Lines: 11679

💛 - Coveralls

@leafty leafty force-pushed the leafty/build-data-connectors branch 2 times, most recently from 6dcb37a to 739e439 Compare September 23, 2024 07:35
leafty added a commit that referenced this pull request Sep 26, 2024
Merges:
* #418
* #407

Merge remote-tracking branch 'origin/feat-add-cloudstorage' into leafty/leafty/feat-namespaced-data-connectors-merge-amalthea-migration
@leafty leafty force-pushed the leafty/build-data-connectors branch 2 times, most recently from 7ecf26e to 7574d13 Compare September 30, 2024 14:24
leafty added 6 commits October 4, 2024 09:25
Moves the `project_slugs` table to `entity_slugs` and `project_slugs_old` to `entity_slugs_old`. This is done to prepare for more entity types to be namespaced.

Note: the sequence migration has been tested:
1. Force migration revision to be `9058bf0a1a12` (before this).
2. Deploy with image tag `renku/renku-data-service:sha-345a1bb` (current `main`) and run `alembic check` and `alembic heads` to verify the DB is before this change.
3. Create some projects.
4. Re-deploy this PR and check that migration revision is `a11752a5afba` (after this).
5. Create more projects.
Introduce namespaced data connectors for Renku 2.0.

Details:
* Remove API routes for `/storages_v2`
* Add API routes for `/data_connectors`
* Add `data_connectors` table
* Drop the `storage_secrets` table
* Add `data_connectors_secrets` table
Fixes deleting storage configuration fields by setting them to `null`.
Adds authorization for data connectors in Renku 2.0.

Details:
* Authz changes to support data connectors

The `authz` schema changes include future support for linking data connectors to projects, but doing so is not implemented in this PR.
Adds the possibility to link a data connector to a project.

Details:
* Add API to link a data connector to a project
* Add API to list projects connected to a data connector
* Add API to list data connectors connected to a project

Note that `read` permissions are cascaded to direct members of the connected project.

* Creating a link from a private data connector to a project requires the `editor` role on both sides of the link.
* Creating a link from a public data connector to a project requires the `editor` role on the project.
* Removing a link requires either the `editor` role on the project or the `owner` role on the data connector.
Add support for saving and managing secrets for data connectors.

Details:
* Add API endpoints to list, update and delete saved secrets for a given data connector.
@leafty leafty force-pushed the leafty/build-data-connectors branch from c5820ef to b772eef Compare October 4, 2024 08:48
leafty added 5 commits October 7, 2024 07:10
…nectors

# Conflicts:
#	components/renku_data_services/project/apispec.py
Fixes two issues with the migrations for data connectors:
1.  The migrations cannot be rolled back after creating a data connector. This was caused by leftover entries in the `entity_slugs` table.
2. The entity slug constraint was not created. The change was not picked up by Alembic; it seems Alembic can only handle `CheckConstraint` creation together with table creation, but not new constraints on an existing table, updates to them, or removing them from a table.
…nectors

# Conflicts:
#	components/renku_data_services/storage/api.spec.yaml
#	components/renku_data_services/storage/apispec.py
#	components/renku_data_services/storage/db.py
#	components/renku_data_services/storage/models.py
Add a background job to migrate "Data Sources" (`/storages_v2`) to "Data Connectors" (`/data_connectors`). Each storage item has a new data connector created and linked to the source project.
@leafty leafty marked this pull request as ready for review October 11, 2024 08:52
@leafty leafty requested a review from a team as a code owner October 11, 2024 08:52
@leafty leafty merged commit ad4abd0 into main Oct 17, 2024
15 checks passed
@leafty leafty deleted the leafty/build-data-connectors branch October 17, 2024 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants